Papers with preference learning
Machine Translation for Low-Resource Languages through Monolingual Data and LLM: A Case Study of English-to-Basque (2026.eacl-srw)
Copied to clipboard
| Challenge: | Existing LLMs do not translate well from English to Basque, but they yield an acceptable performance in the reverse direction. |
| Approach: | They propose to use a Basque monolingual corpora to train an LLM-based MT system . they use 'sovereignty fine tuning' to generate parallel corporata, and then use preference optimization . |
| Outcome: | The proposed system improves translation quality in English-to-Basque direction while requiring limited data for low-resource languages. |
Causal Direct Preference Optimization for Language Model Alignment (2026.findings-eacl)
Copied to clipboard
| Challenge: | Empirical evaluations show that CDPO surpasses DPO-based baselines by achieving unbiased fine-tuning through causal reasoning. |
| Approach: | They propose a framework that incorporates causal inference principles to mitigate the influence of confounders and sharpen the signal of genuine human preferences. |
| Outcome: | The proposed framework preserves the tractability of direct optimization while enhancing robustness to spurious correlations and annotation biases. |
Towards Tool Use Alignment of Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on tool use with LLMs focus on enhancing tool-calling ability of LLM . e.g., LLM should not answer unsafe tool use relevant instructions or insecure tool responses to ensure reliability and harmlessness. |
| Approach: | They propose to use supervised fine-tuning and preference learning to align LLMs with H2A principle for tool use. |
| Outcome: | The proposed model demonstrates that LLMs can generate truthful and helpful responses while remaining harmless. |
Course-Correction: Safety Alignment Using Synthetic Preferences (2024.emnlp-industry)
Copied to clipboard
Rongwu Xu, Yishuo Cai, Zhenhong Zhou, Renjie Gu, Haiqin Weng, Liu Yan, Tianwei Zhang, Wei Xu, Han Qiu
| Challenge: | Recent studies show that large language models generate harmful content, but the potential for generating harmful content is an escalating concern. |
| Approach: | They propose to fine-tune LLMs with preference learning to emphasize the preference for timely course-correction by using an automated pipeline. |
| Outcome: | The proposed model improves course-correction skills without affecting general performance and resists jailbreak attacks. |
Advancing Process Verification for Large Language Models via Tree-Based Preference Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating step-by-step rationales fail to fully utilize the relative merits of intermediate steps, limiting the effectiveness of feedback provided. |
| Approach: | They propose a tree-based preference learning verifier that constructs reasoning trees via a best-first search algorithm and collects step-level paired data for preference training. |
| Outcome: | The proposed approach outperforms existing benchmarks on arithmetic and commonsense reasoning tasks. |
Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have shown that large language models can enhance response richness and coherence, but there is a pressing need to bolster the model’s capacity for diagnostic logic to ensure patient safety. |
| Approach: | They propose an approach termed preference learning from process feedback (PLPF) that integrates the doctor’s diagnostic logic into LLMs. |
| Outcome: | The proposed approach improves the diagnostic accuracy of the baseline model in medical conversations by 17.6%, surpassing the performance of traditional approaches. |
Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback (2022.findings-naacl)
Copied to clipboard
Duy-Hung Nguyen, Nguyen Viet Dung Nghiem, Bao-Sinh Nguyen, Dung Tien Tien Le, Shahab Sabahi, Minh-Tien Nguyen, Hung Le
| Challenge: | a framework to train summarization models with preference feedback is proposed . human-in-the-loop (HITL) allows humans to actively participate in supervising AI systems . |
| Approach: | They propose a framework to train summarization models with preference feedback interactively. |
| Outcome: | The proposed framework improves ROUGE scores and sample-efficiency in active, few-shot and online settings. |
Towards Reward Fairness in RLHF: From a Resource Allocation Perspective (2025.acl-long)
Copied to clipboard
| Challenge: | if rewards are imperfect, they can adversely affect the alignment of large language models (LLMs). |
| Approach: | They propose a bias-agnostic method to address the issue of reward unfairness from a resource allocation perspective without specifically designing for each type of bias . they apply methods Fairness Regularization and Fairness Coefficient to achieve fairness in rewards. |
| Outcome: | The proposed method achieves fairness in rewards while minimizing biases . it can be applied to verification and reinforcement learning scenarios . |
Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent efforts to verify text accuracy provide no guarantees on their correctness . a new method to improve LLMs' verifiability is to use quotes to ground models . |
| Approach: | They propose a method that allows models to quote verbatim statements from trusted sources . they leverage a fast membership inference function to verify text against trusted corpora . |
| Outcome: | The proposed method significantly increases verbatim quotes from high-quality documents by up to 130% relative to base models while maintaining response quality. |
Edit-Wise Preference Optimization for Grammatical Error Correction (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been successful in grammatical error correction (GEC) but their strengths have yet to be fully demonstrated in GEC . |
| Approach: | They propose a method to optimize grammatical errors by assigning higher reward weights to edit tokens during preference optimization. |
| Outcome: | The proposed method outperforms baselines on English and Chinese datasets and achieves state-of-the-art performance. |
Improving Attributed Text Generation of Large Language Models via Preference Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models have been widely adopted in natural language processing, yet they produce unreliable content. |
| Approach: | They propose to model the attribution task as preference learning and introduce an automatic preference optimization framework that synthesizes attribution preference data. |
| Outcome: | The proposed method achieves state-of-the-art citation F1 with higher answer quality than existing methods. |
Synthetic Paths to Integral Truth: Mitigating Hallucinations Caused by Confirmation Bias with Synthetic Data (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to mitigate confirmation bias-induced hallucination in large language models (LLMs) however, they still exhibit issues such as confirmation bias, which remains unexplored in current research. |
| Approach: | They propose a method to mitigate confirmation bias-induced hallucination in large language models by using a synthetic data construction pipeline and direct preference optimization (DPO) training. |
| Outcome: | The proposed method improves response accuracy and reduced hallucination on natural questions open and halubench. |
Beyond Under-Alignment: Atomic Preference Enhanced Factuality Tuning for Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing work evaluates the factuality of large language models on in-domain (ID) datasets and the factuality on out-of-domain datasets. |
| Approach: | They propose a framework that enhances model’s awareness of factuality at the granularity of individual facts and propose 'Atomic Preference Enhanced Factuality Tuning' this framework enhances the model’ s awareness and accuracy of factual information at the level of individual factual facts. |
| Outcome: | The proposed framework improves model performance by an average of on ID and OOD datasets, which is highly effective. |
Learning from Relevant Subgoals in Successful Dialogs using Iterative Training for Task-oriented Dialog Systems (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Task-oriented Dialog (ToD) systems have to solve multiple subgoals to accomplish user goals, whereas feedback is often obtained only at the end of the dialog. |
| Approach: | They propose an iterative training approach that uses subgoals to improve task-oriented dialog systems. |
| Outcome: | The proposed approach improves on a popular ToD benchmark by combining fine-tuning and preference learning steps. |
On Synthetic Data Strategies for Domain-Specific Generative Retrieval (2025.acl-long)
Copied to clipboard
| Challenge: | Generative retrieval models can be used to generate ranked lists of potentially relevant document identifiers for a user query. |
| Approach: | They propose a synthetic data generation strategy for a two-stage training framework that focuses on learning to decode document identifiers from queries and a strategy for mining hard negatives based on initial model's predictions. |
| Outcome: | The proposed model can generate ranked lists of potentially relevant document identifiers for a user query and then refine ranking through preference learning. |
Synthesizing Text-to-SQL Data from Weak and Strong LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | a capability gap exists between open-source and closed-source large language models (LLMs) . the adoption of closed-sourced LLMs introduces concerns pertaining to openness, privacy, and substantial costs. |
| Approach: | They propose a synthetic data approach that combines strong and weak models for error information . they demonstrate the effectiveness of SENSE, a specialized text-to-SQL model . |
| Outcome: | The proposed method enhances the domain generalization of text-to-SQL models and explores the potential of error data supervision through preference learning. |
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning (D18-1)
Copied to clipboard
| Challenge: | Existing preference-based interactive learning methods suffer from high sample complexity, i.e. they need to interact with the oracle for many rounds in order to converge. |
| Approach: | They propose a method which learns from users’ preferences instead of reference summaries by interacting with the oracle for multiple rounds and leveraging active learning, preference learning and reinforcement learning techniques. |
| Outcome: | The proposed method significantly advances the state of the art in both simulation and real-user experiments. |
HEAL: A Hypothesis-Based Preference-Aware Analysis Framework (2025.findings-emnlp)
Copied to clipboard
Yifu Huo, Chenglong Wang, Qiren Zhu, Shunjie Xing, Tong Xiao, Chunliang Zhang, Tongran Liu, JingBo Zhu
| Challenge: | Preference optimization methods like DPO are often evaluated on a single response, overlooking other outputs. |
| Approach: | They propose a Hypothesis-based PrEference-aware AnaLysis Framework that formulates preference alignment as a re-ranking process within hypothesis spaces. |
| Outcome: | The proposed evaluation paradigm re-ranks preference alignment as a reranking process within hypothesis spaces. |
Preference Learning Unlocks LLMs’ Psycho-Counseling Skills (2026.findings-acl)
Copied to clipboard
| Challenge: | Current LLMs struggle to consistently provide effective responses to client speeches due to the lack of supervision from high-quality real psycho-counseling data. |
| Approach: | They propose to use a dataset to evaluate therapists' responses to client speeches using a set of professional and comprehensive principles to evaluate their responses. |
| Outcome: | The proposed model achieves an impressive win rate of 87% against GPT-4o. |
Self-Training Meets Consistency: Improving LLMs’ Reasoning with Consistency-Driven Rationale Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches labeled rationales that produce correct answers as appropriate for training but one measure risks misjudging rationale quality, leading models to learn flawed reasoning patterns. |
| Approach: | They propose a framework that evaluates rationales through follow-up questions and leverages this evaluation to guide its training. |
| Outcome: | The proposed framework improves robustness and correctness of rationales and reasoning abilities compared to previous self-training approaches. |
Cognitive-Level Adaptive Generation via Capability-Aware Retrieval and Style Adaptation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models struggle to adapt content to users with differing cognitive capacities, leading to cognitive misalignment. |
| Approach: | They propose a cognitive-level alignment framework that aligns both knowledge complexity and presentation style with user cognition. |
| Outcome: | The proposed framework aligns knowledge complexity and presentation style with user cognition. |
Speechworthy Instruction-tuned Language Models (2024.emnlp-main)
Copied to clipboard
Hyundong Cho, Nicolaas Jedema, Leonardo Ribeiro, Karishma Sharma, Pedro Szekely, Alessandro Moschitti, Ruben Janssen, Jonathan May
| Challenge: | Current instruction tuned language models are trained on textual preference data and therefore not aligned to speech domain. |
| Approach: | They propose to use radio-industry best practices to prompt and learn speech-based preference data to improve speech-suitability of popular instruction tuned language models. |
| Outcome: | The proposed methods achieve the best win rates in head-to-head comparisons, resulting in preferred or tied to the base model in 76.2% of comparisons on average. |
STeCa: Step-level Trajectory Calibration for LLM Agent Learning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work focuses on behavior cloning from expert demonstrations or preference learning through exploratory trajectory sampling, but these methods often struggle to address long-horizon tasks where suboptimal actions accumulate step by step, causing agents to deviate from correct task trajectories. |
| Approach: | They propose a framework for LLM-based agent learning that identifies suboptimal actions through a step-level reward comparison during exploration and constructs calibrated trajectories using LLM reflection. |
| Outcome: | The proposed framework outperforms existing methods in long-horizon tasks where suboptimal actions accumulate step by step, causing agents to deviate from correct task trajectories. |
Preference Consistency Matters: Enhancing Preference Learning in Language Models with Automated Self-Curation of Training Corpora (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to address inconsistencies in preference learning datasets rely on heuristics to achieve alignment. |
| Approach: | They propose a method that preprocesses annotated datasets by leveraging proxy models trained directly on them to detect and select consistent annotations. |
| Outcome: | The proposed method shows performance improvements of up to 33% across learning algorithms and proxy capabilities. |
MuCAL: Contrastive Alignment for Preference-Driven KG-to-Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for KG-to-text generation are limited by the availability of reliable preference data. |
| Approach: | They propose to use a multilingual KG/Text alignment model to generate preference data using three LLMs by ranking candidates and applying Direct Preference Optimization (DPO) on these preferences. |
| Outcome: | The proposed model achieves robust cross-modal retrieval across multiple languages and difficulty levels. |
Reward Model Perspectives: Whose Opinions Do Reward Models Reward? (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that reward models are poorly aligned with demographic groups and can reward harmful stereotypes. |
| Approach: | They propose a framework for measuring the alignment of opinions captured by RMs . they also investigate the extent to which RM's demonstrate sociodemographic biases a . |
| Outcome: | The proposed framework measures the alignment of opinions captured by RMs . it shows that RM models are poorly aligned with several demographic groups . the findings highlight the need for more careful consideration of RM behavior in model alignment . |
V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing large vision-language models suffer from hallucination due to over-reliance on the Large Language Model (LLM) backbone. |
| Approach: | They propose a method to improve visual context learning by using a large-scale preference learning algorithm to improve hallucination. |
| Outcome: | The proposed method improves on human-annotated hallucination datasets. |
Reward Generalization in RLHF: A Topological Perspective (2025.findings-acl)
Copied to clipboard
Tianyi Alex Qiu, Fanzhi Zeng, Jiaming Ji, Dong Yan, Kaile Wang, Jiayi Zhou, Yang Han, Josef Dai, Xuehai Pan, Yaodong Yang
| Challenge: | Existing alignment methods share a common topology of information flow, but their alternatives have not been thoroughly explored. |
| Approach: | They propose a theory of reward generalization in reinforcement learning from human feedback . they propose induced Bayesian networks to model the impact of dataset topologies on reward generalisation . |
| Outcome: | The proposed method achieves an average win rate of 65% on three NLP tasks. |
SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for preference optimization of large language models use pairs of positive and negative samples, but the quality of positive samples may become similar during training, complicating preference learning. |
| Approach: | SeaPO introduces error types commonly occurring in large language models to improve preference learning. |
| Outcome: | SeaPO introduces error types into model Preference Optimization to improve model performance . negative samples are more erroneous than positive samples, and preference-based training mitigates errors . |
Annotation-Efficient Language Model Alignment via Diverse and Representative Response Texts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | obtaining large amount of preference annotations is difficult in many applications . obtaining a large amount is difficult, so a preference dataset needs limited annotation budget . |
| Approach: | They propose annotating preference over a subset of responses that maximizes diversity and representativeness from available responses and then annotates preference over the selected ones. |
| Outcome: | The proposed method outperforms baselines with the same annotation budget. |
World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches focus on action selection or use pre-trained models as world models to enhance planning capabilities. |
| Approach: | They propose a new learning framework that optimizes state prediction and action selection through preference learning. |
| Outcome: | The proposed method outperforms existing methods and GPT-4o on VoTa-Bench and Qwen2-VL (7B), LLaVA-1.6 (7B) and LLama-3.2 (11B). |
Thinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal Refinement (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent research indicates that large language models (LLMs) have demonstrated remark-able capabilities in various programming-related domains, such as code generation and code refinement. |
| Approach: | They propose a framework that combines exploration with refinement to reduce test-time computation overhead. |
| Outcome: | The proposed framework outperforms SOTA and AgentCoder on humanEval and MBPP benchmarks while reducing test-time computation overhead and scalability. |
From Lists to Emojis: How Format Bias Affects Model Alignment (2025.acl-long)
Copied to clipboard
| Challenge: | Format biases in reinforcement learning from human feedback are underexplored . despite its effectiveness, RLHF faces challenges, including policy and regulatory constraints . |
| Approach: | They extend the study of preference biases beyond verbosity bias to a wider range of format biase . they show that with a small amount of biased data, they can inject significant bias into the reward model . |
| Outcome: | The proposed approach can be easily exploited by large language models to achieve higher rankings on popular benchmarks like AlpacaEval and LMSYS Chatbot Arena. |
Subtle Errors in Reasoning: Preference Learning via Error-injected Self-editing (2025.acl-long)
Copied to clipboard
Kaishuai Xu, Tiezheng Yu, Wenjun Hou, Yi Cheng, Chak Tou Leong, Liangyou Li, Xin Jiang, Lifeng Shang, Qun Liu, Wenjie Li
| Challenge: | Existing studies to improve mathematical ability typically involve applying preference learning to step-wise solution pairs, but they overlook critical subtle errors. |
| Approach: | They propose a preference learning framework that injects predefined subtle errors into pivotal tokens to construct hard pairs for error mitigation. |
| Outcome: | Extensive experiments show that the proposed framework improves on Qwen2-7B-Instruct and MATH with 4.5K training samples. |
Expectation Preference Optimization: Reliable Preference Estimation for Improving the Reasoning Capability of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Pairwise preference optimization is used to improve supervised fine-tuning performance of large language models. |
| Approach: | They propose an algorithm that takes pairs of sample groups instead of single samples for preference learning. |
| Outcome: | The proposed algorithm outperforms baseline methods on reasoning benchmarks. |
Edit-Aware Reward Modeling for Chinese Grammatical Error Correction (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work has applied reinforcement learning with rule-based rewards to grammatical error correction tasks, but these methods fail to capture fine-grained quality distinctions among correction candidates. |
| Approach: | They propose an Edit-Aware Reward Model that explicitly incorporates edit-awareness into preference learning for CGEC. |
| Outcome: | The proposed model outperforms rule-based models on CGEC and other NLP tasks by 5.41 and 1.80 points. |
Verified Critical Step Optimization for LLM Agents (2026.findings-acl)
Copied to clipboard
| Challenge: | Critical Step Optimization (CSO) focuses preference learning on verified critical steps where alternative actions demonstrably flip task outcomes from failure to success. |
| Approach: | They propose a method which focuses preference learning on verified critical steps where alternative actions demonstrably flip task outcomes from failure to success. |
| Outcome: | The proposed method outperforms the existing methods on GAIA-Text-103 and XBench-DeepSearch while requiring supervision at only 16% of trajectory steps. |
Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback (2026.findings-acl)
Copied to clipboard
Siddhant Arora, Jinchuan Tian, Jiatong Shi, Hayato Futami, Yosuke Kashiwagi, Emiru Tsunoo, Shinji Watanabe
| Challenge: | Existing studies on reinforcement learning from human or AI feedback have focused on semantic rewards at the utterance level. |
| Approach: | They propose a multi-reward RLAIF framework for speech-in/speech-out dialogue systems . they combine semantic, audio-quality, and emotion-consistency rewards . |
| Outcome: | The proposed framework improves speech-in/speech-out dialogue system quality . it combines semantic, audio-quality, and emotion-consistency rewards . the proposed framework is available to download from the cdc. |